Intelligent decision support system

ABSTRACT

Various embodiments are provided for implementing intelligent decision support system in a computing environment by a processor. Data of historical decisions may be collected and examples of decisions by domain experts may be generated. One or more machine learning models may be generated using different splits of the historical data and the annotated data. The one or more machine learning models may be combined and used to generate ensemble machine learning models that generate recommendations for the decisions. Users interact with a user interface displaying the data, recommendations, reasons for recommendations and a conversational dialog system for querying about the data, recommendations and guidance for decision making.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to India Patent Application No.201841021354 filed on Jun. 7, 2018 titled “Interactive Decision MakingSupport System and Method Thereof.”

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates in general to computing systems, and moreparticularly, to various embodiments for implementing intelligentdecision support system using a computing processor.

Description of the Related Art

In today's society, consumers, businesspersons, educators, and otherscommunicate over a wide variety of mediums in real time, across greatdistances, and many times without boundaries or borders. The advent ofcomputers and networking technologies have made possible the increase inthe quality of life while enhancing day-to-day activities. For example,processing devices, with the advent and further miniaturization ofintegrated circuits, have made it possible to be integrated into a widevariety of devices. As great strides and advances in technologies cometo fruition, these technological advances can be then brought to bear ineveryday life. For example, the vast amount of available data madepossible by computing and networking technologies may then assist inimprovements to quality of life and appropriate living conditions.

SUMMARY OF THE INVENTION

Various embodiments are provided for implementing intelligent decisionsupport systems in a computing environment. The intelligent decisionsupport system may collect and use data about historical decisions(which may have been performed by a domain expert) in those domains(e.g., historical data) and/or have domain experts generate examples ofgood decisions in those domains (annotated data). One or more machinelearning models may be built using different splits of the historicaldata and the annotated data to generate recommendations for decisionmaking. The machine learning models may be combined to build ensemblemachine learning models to generate recommendations for decision making.

In addition to the foregoing exemplary method embodiment, otherexemplary system and computer product embodiments are provided andsupply related advantage.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict embodiments of the invention and are not therefore to beconsidered to be limiting of its scope, the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 is a block diagram depicting an exemplary computing nodeaccording to an embodiment of the present invention;

FIG. 2 is an additional block diagram depicting an exemplary cloudcomputing environment according to an embodiment of the presentinvention;

FIG. 3 is an additional block diagram depicting abstraction model layersaccording to an embodiment of the present invention;

FIG. 4 is an additional block diagram depicting an exemplary functionalrelationship between various aspects of the present invention;

FIG. 5A is a diagram depicting a user interface display for anadministrator using an intelligent decision support system in accordancewith aspects of the present invention;

FIG. 5B is a diagram depicting a user interface display for a user usingan intelligent decision support system in accordance with aspects of thepresent invention;

FIG. 5C is a diagram depicting a user interface display for a senioradministrator using an intelligent decision support system in accordancewith aspects of the present invention;

FIG. 5D is a diagram depicting a user interface display for collectingfeedback using an intelligent decision support system in accordance withaspects of the present invention;

FIG. 5E is a diagram depicting an integrated dialog system using theintelligent decision support system in accordance with aspects of thepresent invention;

FIG. 6 is a diagram depicting an integrated dialog system complementingthe intelligent decision support system in accordance with aspects ofthe present invention;

FIG. 7A-7C are block diagrams depicting operations for generatingnatural language reasons (“NLR”) for explaining automatically generatedmachine learning recommendations from an ensemble of classifiers inaccordance with aspects of the present invention;

FIG. 8A is an additional block diagram depicting an intelligent decisionsupport system in accordance with aspects of the present invention;

FIG. 8B is a flowchart diagram depicting an exemplary method for usingan intelligent decision support system in a computing environment inaccordance with an embodiment of the present invention;

FIG. 9 is a diagram depicting a user interface display providingrecommendations using an intelligent decision support system inaccordance with aspects of the present invention;

FIG. 10 is an additional block diagram depicting an intelligent decisionsupport system in accordance with aspects of the present invention;

FIG. 11 is an additional diagram depicting an intelligent decisionsupport system in accordance with aspects of the present invention;

FIG. 12 is a flowchart diagram depicting an exemplary method forimplementing intelligent decision support system in a computingenvironment in accordance with an embodiment of the present invention;and

FIG. 13 is a flowchart diagram depicting an additional exemplary methodfor implementing intelligent decision support system in a computingenvironment in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

As a preliminary matter, computing systems may include large scalecomputing called “cloud computing,” in which resources may interactand/or be accessed via a communications system, such as a computernetwork. Resources may be software-rendered simulations and/oremulations of computing devices, storage devices, applications, and/orother computer-related devices and/or services run on one or morecomputing devices, such as a server. For example, a plurality of serversmay communicate and/or share information that may expand and/or contractacross servers depending on an amount of processing power, storagespace, and/or other computing resources needed to accomplish requestedtasks. The word “cloud” alludes to the cloud-shaped appearance of adiagram of interconnectivity between computing devices, computernetworks, and/or other computer related devices that interact in such anarrangement.

Additionally, dialog systems of a computing system can play a key rolein the functioning of an entity, such as a business, government, groupor other organization. For example, many critical decisions may resultfrom discussions in chat systems, or chat-like conversation systems orchatbots. A chatbot may be an operation which conducts a dialog orconversation, audible, visual, and/or via textual methods. Organizationsmay seek to capture and analyze these decisions to make variousimprovements to a structure of the organization.

Moreover, a decision support system may be provided for improving theability of a human in making decisions assisted by machine learninggenerated recommendations for the decisions. For such decision supportsystems, it is critical that an entity (e.g., a human) trust therecommendations and for the decision support system to help generatethis trust, such as, for example, in human resource (“HR”)-relateddomains, where there is a special need to understand decisions,guidance, reasoning, etc., so that entity decision makers can trust thedecision support recommendations.

Accordingly, a need exists for a decision support system that providesan integrated user experience where the recommendations are coupled witha natural language conversational dialog in an integrated user interface(“UP”) to support the decision making.

Thus, the present invention provides an intelligent computing systemthat implements an intelligent decision support system in a computingenvironment. Data of historical decisions may be collected and examplesof decisions by one or more domain experts may be generated. One or moremachine learning models may be generated using different splits of thehistorical data and the annotated data. The one or more machine learningmodels may be combined and used to generate ensemble machine learningmodels that generate recommendations for the decisions. Users interactwith a user interface displaying the data, recommendations, reasons forrecommendations and a conversational dialog system for querying aboutthe data, recommendations and guidance for decision making.

In an additional aspect, the intelligent decision support system maycollect similar data according to historical data and annotated data.One or more prediction models may be generated using different splits ofthe similar data. The one or more prediction models may be combined andused to generate a decision model using one or more machine learningoperations.

In an additional aspect, the intelligent decision support system may bean intelligent/cognitive decision support system that assistssupervisors/managers of an entity with compensation decisions byrecommending one or more actions for one or more selectedusers/employees according to a complete profile/understanding of theemployee such as, for example, job performance, skills and/or sets ofskills, talents, abilities, competitiveness of the user/employees'compensation, compensation history, promotions/advancements, prioremployments, and/or other historical data pertaining to the user.

The intelligent decision support system may include a machine learningcomponent/mechanism to generate one or more recommendations to increasethe intelligence, reasoning, and effectiveness of various users (e.g.,administrators, supervisors, managers, etc.) making decisions that mayinfluence one or more outcomes such as, for example, increasingcompetitiveness of compensation, investing in skilled employees,reducing attrition of experienced and/or highly-skilled employees.

As an additional aspect, the present invention provides for anintelligent decision support system having an integrated user interface(“UP”) for one or more decisions (e.g., HR based decisions) with machinelearning recommendations, evidence, and reasons supporting therecommendations. A conversational dialog/chatbot in the intelligentdecision support system may be used to engage in a dialog (e.g.,question/answer dialog) pertaining to one or more of the decisions,recommendations, guidance, etc.

In one aspect, the intelligent decision support and UI system mayinclude/provide the following features and/or functionality. First, oneor more recommendations for decisions may be generated from a machinelearning model. Second, one or more analytics that dynamically analyzehuman decisions may be used to provide various types ofdecisions/suggestions. Third, analytics suggesting evidence and reasonsfor each recommendation may also be provided. Fourth, a natural languageconversational chatbot may be used for answering (e.g., in real-time)any user queries pertaining to information related to the decisions,reasons behind recommendations, guidance for decision making, etc.Fifth, the intelligent decision support and UI system mayinclude/provide the ability for selected personnel/managers to enterand/or validate various levels of skills and expertise levels for thosepersons the selected personnel/managers have direct responsibility overand to aid in the decision making. Sixth, the intelligent decisionsupport and UI system may group employee attributes to reflect one ormore priorities of an entity (e.g., salary program objectives) in termsof directions for the corresponding decisions.

In an additional aspect, the present invention provides for anintelligent decision support system using one or more machine learningmodels (e.g., decision trees or an ensemble of decision trees) that mayoutput, for each data point, a class label. However, some applicationdomains (e.g., HR-related domains such as, for example, compensationdecision making based on varied employee data) need an ability toexplain and interpret a chain of reasoning used by artificialintelligence “AI” and/or machine learning models to arrive atrecommendations. Thus, the present invention provides one or moremachine learning models (e.g., decision tree models) with a chain ofreasoning (a complete machine learning path in the case of decisiontrees) followed by the machine learning models for every data point togenerate recommendations, which may be in a user consumable format(e.g., natural language format). In this way, a user is enabled tounderstand the decision (e.g., a class label) as well as the path/rulesfollowed to reach the decision in a natural language format.

In specific instances, where an ensemble of decision trees is used togenerate recommendations, the present invention provides a computingsystem that generates a full/complete path taken for each input datapoint across an ensemble of machine learning models to generate thefinal recommendation (decision). In one aspect, the decision path foreach of the decision trees in the ensemble of machine learning modelsmay be determined/computed. For each decision tree in the ensemble ofmachine learning models, the present invention may linearize a completedecision tree into a set of machine learning rules where each setcorresponds to a single path taken from a root node to the leaf(decision) node of the decision tree.

For each input data point, the present invention may compare the valuesof the attributes of the data point against each of the machine learningrules to identify a single unique path corresponding to the data pointfrom the root node to the leaf node(s) of the decision tree. The single,unique path may be a set of individual machine learning rules. For eachmachine learning rule in the single, unique path, the present inventionmay apply an appropriate natural language rule that factors in themachine learning rule as well as an eventual decision of the machinelearning path. From an ensemble of decision trees, the present inventionmay apply a voting approach to select a set of relevant decision trees.The present invention may concatenate the natural language reasonsacross all selected decision trees to generate a holistic set of reasonsfor each final recommendation (e.g., decision).

In an additional aspect, unlike most chatbots that are only based onstatic, unstructured content, for a selected domain (e.g., a HR domain),the present invention provides one or more customized/personalizeddialog systems/chatbots that are enabled to answer one or morequeries/questions against both structured and unstructured data andprovide both customized/personalized information specific to a user(e.g., an employee) as well as generalized information for differentgroups of users (e.g., employees). For example, a chatbot as part of anintelligent decision support system (e.g., an intelligent HR decisionsupport system) may provide personalized information about specificemployees as well as generic guidance about an employee programapplicable to all employees.

In one aspect, the intelligent decision support system provides as inputa structured database of user information (e.g., a structured HRdatabase of employee information), guidelines and information of anentity (e.g., General HR guidelines and information) such as, forexample compensation planning. The intelligent decision support systemmay provide as output, using a dialog system/chatbot (e.g., aconversational chatbot), one or more selected users that areautomatically identified (e.g., employees identified from a query). Theintelligent decision support system may detect the intent of thecommunication in the chatbot and fetches the information to answer thequery of the chatbot. Thus, the present invention combines structuredand unstructured data, customized/personalized information, andinformation in the chatbot (e.g., a HR chatbot). The present inventionmay also provide a customized/personalized chatbot that contextuallydetects a user relating to a query (e.g., an employee being asked aboutto) so as to retrieve/fetch relevant information pertaining to the user.The data relating to the query may be retrieved/fetched from a selecteddatabase having a domain knowledge (e.g., an employee database or HRknowledgebase).

In one aspect, the intelligent decision support system may use one ormore machine learning models for generating compensation suggestions toone or more users (e.g., managers). To build accurate machine learningmodels, training data may be used. The training data sources may behistorical data (e.g., previous historical data relating to compensationdecisions) and/or annotated data (e.g., data annotated by a domainexpert that may be coded as examples.

The historical data may be a collection of historical decisions alongwith a variety of data pertaining to decision making. The historicaldata may be analyzed, selected, and sampled according to the context andneeds/requirements of the decision making. A degree of importance ofdifferent attributes may be determined and analyzed since the degree ofimportance of different attributes may change over time. Moreover, thehistorical data may include data attributes for one or more users (e.g.,employees for HR use cases) at the time of historical decision making,since the data attributes may change over time.

The annotated data may be a collection of examples/samples of decisionsmade by one or more domain experts, which may be for the intended/solepurpose of improving the machine learning models. In one aspect, theannotated data by domain experts may be generated in anenvironment/setup that is as close as possible to the actual decisionmaking environment/setup. The domain experts may be carefully sampledfrom a plurality of domain experts in decision making context (e.g., HRcompensation experts) as well as sampling of decision makers.

In an additional aspect, the mechanism of the illustrated embodimentsmay utilize machine learning for one or more applications for one ormore entities and domains (e.g., various industries and domains).Supervised machine learning operations require labelled data. However,it is often costly and time consuming to generate labeled data for aparticular application. Often, labeled data sets may be available thatmay not be exact or precise/perfect fits for a given application. Forexample, for a HR decision making scenario, though data about historicalHR decisions may be available, such data may not be an appropriateand/or perfect fit for a current decision process since thecriteria/context of the decision process now may have shifted in anorganization. If more precise, appropriate, and/or specific examplesthat fit the application/decision making context can beharvested/separated from these other data sets, the data may boostand/or optimize accuracy of a machine learning model. Thus, theintelligent decision support system may provide as input an annotated,labeled, data set matching a machine learning requirement and/or otherlabeled data sets that are not a perfect match for a current machinelearning requirement.

The intelligent decision support system may provide as output aselection of specific examples from other data sets that are best fits(e.g., optimal fit) for the current machine learning requirement, thusboosting accuracy of machine learning modeling. For example, theintelligent decision support system may select data samples from a dataset B based on similarity to data set A for use by a specific machinelearning application that is closely aligned with data set A. Suchoperations may be used by combining historical data about historicaldecisions (e.g., manager decisions) with current annotation data (from adomain expert). The intelligent decision support system may identify adegree of matching between a data point in a data set B with a datapoint in a data set A. A sample from data set B may be selected that is“similar” to data set A. A machine learning model may be trained bycombining data set A and the selected sample from data set B.

Additional aspects of the present invention and attendant benefits willbe further described, following.

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security parameters, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 1, a schematic of an example of a cloud computingnode is shown. Cloud computing node 10 is only one example of a suitablecloud computing node and is not intended to suggest any limitation as tothe scope of use or functionality of embodiments of the inventiondescribed herein. Regardless, cloud computing node 10 is capable ofbeing implemented and/or performing any of the functionality set forthhereinabove.

In cloud computing node 10 there is a computer system/server 12, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 1, computer system/server 12 in cloud computing node 10is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 12 may include, but are not limitedto, one or more processors or processing units 16, a system memory 28,and a bus 18 that couples various system components including systemmemory 28 to processor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,system memory 28 may include at least one program product having a set(e.g., at least one) of program modules that are configured to carry outthe functions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in system memory 28 by way of example, and not limitation,as well as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 2, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or other type of computer systems 54N (e.g., a smart watch,biometric sensor, health state monitoring computer system, etc.) maycommunicate. Nodes 10 may communicate with one another. They may begrouped (not shown) physically or virtually, in one or more networks,such as Private, Community, Public, or Hybrid clouds as describedhereinabove, or a combination thereof. This allows cloud computingenvironment 50 to offer infrastructure, platforms and/or software asservices for which a cloud consumer does not need to maintain resourceson a local computing device. It is understood that the types ofcomputing devices 54A-N shown in FIG. 2 are intended to be illustrativeonly and that computing nodes 10 and cloud computing environment 50 cancommunicate with any type of computerized device over any type ofnetwork and/or network addressable connection (e.g., using a webbrowser).

Referring now to FIG. 3, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 2) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 3 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Device layer 55 includes physical and/or virtual devices, embedded withand/or standalone electronics, sensors, actuators, and other objects toperform various tasks in a cloud computing environment 50. Each of thedevices in the device layer 55 incorporates networking capability toother functional abstraction layers such that information obtained fromthe devices may be provided thereto, and/or information from the otherabstraction layers may be provided to the devices. In one embodiment,the various devices inclusive of the device layer 55 may incorporate anetwork of entities collectively known as the “internet of things”(IoT). Such a network of entities allows for intercommunication,collection, and dissemination of data to accomplish a great variety ofpurposes, as one of ordinary skill in the art will appreciate.

Device layer 55 as shown includes sensor 52, actuator 53, “learning”thermostat 56 with integrated processing, sensor, and networkingelectronics, camera 57, controllable household outlet/receptacle 58, andcontrollable electrical switch 59 as shown. Other possible devices mayinclude, but are not limited to various additional sensor devices,networking devices, electronics devices (such as a remote controldevice), additional actuator devices, so called “smart” appliances suchas a refrigerator or washer/dryer, and a wide variety of other possibleinterconnected objects.

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provides cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provides pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and, in the context of the illustratedembodiments of the present invention, various workloads and functions 96for using an intelligent decision support system. In addition, theworkloads and functions 96 for using an intelligent decision supportsystem may include such operations as data analytics, data analysis, andas will be further described, notification functionality. One ofordinary skill in the art will appreciate that the workloads andfunctions 96 for using an intelligent decision support system may alsowork in conjunction with other portions of the various abstractionslayers, such as those in hardware and software 60, virtualization 70,management 80, and other workloads 90 (such as data analytics processing94, for example) to accomplish the various purposes of the illustratedembodiments of the present invention.

Turning now to FIG. 4, a block diagram depicting exemplary functionalcomponents of an intelligent decision support system 400 according tovarious mechanisms of the illustrated embodiments is shown. FIG. 4illustrates workloads and functions for implementing the intelligentdecision support system 400 in a computing environment. As will be seen,many of the functional blocks may also be considered “modules” or“components” of functionality, in the same descriptive sense as has beenpreviously described in FIGS. 1-3. With the foregoing in mind, themodule/component blocks may also be incorporated into various hardwareand software components of a system for implementing the intelligentdecision support system 400 in accordance with the present invention.Many of the functional blocks of the intelligent decision support system400 may execute as background processes on various components, either indistributed computing components, or on the user device, or elsewhere.

Computer system/server 12 is again shown, incorporating processing unit16 (and memory 28 of FIG. 1 and not shown in FIG. 4 for illustrativeconvenience) to perform various computational, data processing and otherfunctionality in accordance with various aspects of the presentinvention.

The intelligent decision support system 400 (e.g., artificialintelligence “AI” service) may include an intelligent decision supportservice 402 and a dialog system 404. The intelligent decision supportservice 402 may include a profile component 410, a recommendationcomponent 420, an analysis component 430, a machine learning component440, and/or a feedback component 450.

The intelligent decision support service 402 and the dialog system 404may each be associated with and/or in communication with each other, byone or more communication methods, such as a computing network. In oneexample, the intelligent decision support service 402 and the dialogsystem 404 may be controlled and/or used by an owner, user/customer(e.g., a manager of a business and/or employed by the business), ortechnician/administrator associated with the computer system/server 12.

In one aspect, the computer system/server 12 may provide virtualizedcomputing services (i.e., virtualized computing, virtualized storage,virtualized networking, etc.) to the intelligent decision supportservice 402 and the dialog system 404. More specifically, the computersystem/server 12 may provide virtualized computing, virtualized storage,virtualized networking and other virtualized services that are executingon a hardware substrate.

The profile component 410 may provide a collection of data (e.g.,historical data) pertaining to each user. In one aspect, the data mayinclude user data for a particular employee, manager/leader, and/orexecutive for an entity (e.g., a business, academic institution,organization, governmental institution, etc.). For example, user datafor an employee may include a holistic view of the employees'performances, skills, compensation, salary competitiveness, attritionrisk, and/or employment/career potential.

The machine learning component 440, along with the recommendationcomponent 420 may collect similar data according to historical data andannotated data, which may be stored in the profile component 410.

The machine learning component 440 may learn, train, and/or generate oneor more prediction models using different splits of the similar data.The machine learning component 440, along with the recommendationcomponent 420, may combine the one or more prediction models to generatea decision model using one or more machine learning operations. Themachine learning component 440, along with the recommendation component420, using the decision models, may provide one or more recommendationspertaining to a selected user (e.g., an employee) that may prioritizeone or more selected factors pertaining to employment (e.g., salaryprioritization) along with providing supporting analytics, reasons,and/or evidence that supports the reasons.

The analysis component 430 may analyze data and each of the decisionsfor a selected user (e.g., a user). For example, the analysis component430 may provide analytics on decisions for a manager and/or a teamwithin a business to assist a manager to analyze salary and/or salaryprogram objectives. Thus, the analysis component 430 may identify one ormore reasons for each of the decisions for a selected user.

Thus, the machine learning component 440, along with the recommendationcomponent 420, may assist with providing decisions (e.g., compensationdecision) and support of each decision such as, for example, byrecommending personalized actions based on a complete understanding ofeach user/employee based on performance, compensation competitiveness,skills and career potential.

The machine learning component 440, along with the recommendationcomponent 420, may combine the structured and unstructured data from oneor more data sources and customize communications in the dialog system404 according to the structured and unstructured data.

The machine learning component 440, along with the recommendationcomponent 420, may interpolate the historical data and the annotateddata using an ensemble of classifiers. The machine learning component440, along with the recommendation component 420, may select theannotated data from a data set based on a degree of similarity with oneor more alternative data sets. The machine learning component 440, alongwith the recommendation component 420, may explain one or morerecommendations from the one or more prediction models according to anatural language operation.

In an additional aspect, the machine learning component 440, along withthe recommendation component 420, may 1) collect the historical data, 2)collect the annotated data from one or more domain experts, 3) derivethe different splits of both the historical data and the annotated data,and/or 4) determine one or more features from one or more attributes ofa selected entity from similar data fields of the different splits.

In one aspect, the machine learning component 440 may be initiated toperform one or more machine learning operations to perform a semanticanalysis, train a classifier, learn one or more machine learning rules,learn contextual data associated with the dialog system, learn and trainthe one or more prediction models using the historical data and theannotated data, generate one or more recommendations or predictions fromthe one or more prediction models, and assist with engaging incommunication using the dialog system, or perform a combination thereof.

The machine learning component 440 may perform one or more machinelearning operation and learn information based on the feedback collectedfrom one or more users via the feedback component 450. For example, oneor more users may engage the dialog system and the feedback component450 may provide feedback to assist the analysis component 430 with oneor more reasons, evidences, or justification for identifying thepositive and/or negative sentiment of a user. The feedback component 450may store the feedback information in a database/memory and may use thefeedback data to learn.

The machine learning component 440 may perform one or more machinelearning operations such as, for example, using natural languageprocessing (NLP) and artificial intelligence (AI) for performing one ormore operations as described herein and for engaging in communicationwith a user via the dialog system 404. The instances of the NLP or AImay include an instance of IBM® Watson®. (IBM® and Watson® aretrademarks of International Business Machines Corporation).

The machine learning component 440 may perform a machine learningoperation for training and learning one or more machine learning modelsand also for learning, applying inferences, and/or reasoning pertainingto one or more users. For example, the machine learning component 440may data to train a classifier of the recommendation component 420and/or analysis component 430.

In one aspect, the learning component 440 may apply one or moreheuristics and machine learning based models using a wide variety ofcombinations of methods, such as supervised learning, unsupervisedlearning, temporal difference learning, reinforcement learning and soforth. Some non-limiting examples of supervised learning which may beused with the present technology include AODE (averaged one-dependenceestimators), artificial neural network, backpropagation, Bayesianstatistics, naive bays classifier, Bayesian network, Bayesian knowledgebase, case-based reasoning, decision trees, inductive logic programming,Gaussian process regression, gene expression programming, group methodof data handling (GMDH), learning automata, learning vectorquantization, minimum message length (decision trees, decision graphs,etc.), lazy learning, instance-based learning, nearest neighboralgorithm, analogical modeling, probably approximately correct (PAC)learning, ripple down rules, a knowledge acquisition methodology,symbolic machine learning algorithms, sub symbolic machine learningalgorithms, support vector machines, random forests, ensembles ofclassifiers, bootstrap aggregating (bagging), boosting (meta-algorithm),ordinal classification, regression analysis, information fuzzy networks(IFN), statistical classification, linear classifiers, fisher's lineardiscriminant, logistic regression, perceptron, support vector machines,quadratic classifiers, k-nearest neighbor, hidden Markov models andboosting. Some non-limiting examples of unsupervised learning which maybe used with the present technology include artificial neural network,data clustering, expectation-maximization, self-organizing map, radialbasis function network, vector quantization, generative topographic map,information bottleneck method, IBSEAD (distributed autonomous entitysystems based interaction), association rule learning, apriorialgorithm, eclat algorithm, FP-growth algorithm, hierarchicalclustering, single-linkage clustering, conceptual clustering,partitional clustering, k-means algorithm, fuzzy clustering, andreinforcement learning. Some non-limiting examples of temporaldifference learning may include Q-learning and learning automata.Specific details regarding any of the examples of supervised,unsupervised, temporal difference or other machine learning described inthis paragraph are known and are considered to be within the scope ofthis disclosure.

Turning now to FIGS. 5A-5E, various displays (e.g., a userinterface/graphical user interface “GUI”) of a dialog system aredepicted using the intelligent decision support system 400 of FIG. 4. Inone aspect, many of the functional blocks previously described in FIGS.1-4 may be applied and used for executing one or more operations and/orfunctionality described in FIGS. 5A-5E. Repetitive description of likeelements, components, modules, services, applications, and/or functionsemployed in other embodiments described herein is omitted for sake ofbrevity.

Turning now to FIG. 5A, diagram 500 illustrates a display 502 (e.g., auser interface/graphical user interface “GUI”) for an administrator(e.g., a home screen for a manager of a business) using the intelligentdecision support system 400 of FIG. 4. For example, the display 500 mayindicate a budget, a percentage of the budget used, a remaining amountof the budget, direct reportee/employee names and data, a dynamicanalytics widget (e.g., products/services) (e.g., for team analytics), apercentage of planning completed, a name of anadministrator/supervisor/manger, an entry section for one or moredecisions, data filter options, and/or a plurality of other customizablefields, attributes, or desired display features for using theintelligent decision support service 402 and the dialog system 404 ofFIG. 1.

Turning now to FIG. 5B, diagram 525 illustrates displays 504 and 506(e.g., a user interface/graphical user interface “GUI”) for one or moreusers (e.g., a home screen for an employee of a business) using theintelligent decision support system 400 of FIG. 4. For example, usingthe intelligent decision support system 400 of FIG. 4, the displays 504and 506 are provided for one or more selected users such as, forexample, employee John Doe 1 and John Doe 2. That is, the intelligentdecision support system 400 may provide the displays 504 and 506 with auser profile (e.g., a detailed employee profile for John Doe 1 and/orJohn Doe 2), one or more recommendations, and/or one or morereasons/justifications for the one or more recommendations. For example,the intelligent decision support system 400 may recommend that John Doe1 is a top contributor and an “employee that possess skills which arescarce in the market” and “the current PMR (e.g., a salarycompetitiveness measure) is significantly low”. Thus, the intelligentdecision support system 400 (e.g., artificial intelligence “AI” service)recommends a high priority for John Doe 1 to have a compensation (e.g.,salary/wage) increase. Alternatively, the intelligent decision supportsystem 400 may recommend that John Doe 2 is a top contributor and thecurrent PMR is significantly low. Thus, the intelligent decision supportsystem 400 recommends a medium priority for John Doe 2 to have acompensation (e.g., salary/wage) increase.

Turning now to FIG. 5C, diagram 535 illustrates displays 540 and 550(e.g., a user interface/graphical user interface “GUI”) for one or moresenior administrator (e.g., a GUI displaying compensation data“compensation view” and budget data “budget view” for a senior managerof a business) using the intelligent decision support system 400 of FIG.4. For example, using the intelligent decision support system 400 ofFIG. 4, the display 540 depicts the ability to select/view one or moreemployees that may be assigned under a selected administrator (e.g.,manager) and display corresponding user profile data (e.g.,compensation) and recommendations. The display 550 depicts the progressof one or more reporting managers and progress of the user (e.g.,user/manager) along with a selected group of the user (e.g., budgettotals of the team) for an overall budget. The using the displays 540and 550 may also enable the selected administrator to submit/approve oneor more suggested recommendations (e.g., submit/approve a budget).

Turning now to FIG. 5D, diagram 545 illustrates displays 547 (e.g., auser interface/graphical user interface “GUI”) for collecting feedbackusing the intelligent decision support system 400 of FIG. 4. Forexample, using the intelligent decision support system 400 of FIG. 4,the display 547 depicts a series of questions and associatedoptions/answers for collecting feedback. For example, the display 547may collect reasons, evidences, or justification for rating an overallexperience using the intelligent decision support system 400, reasons,evidences, or justification for rating for assistance in determining adecision (e.g., a decision to provide compensation raises), rating theintelligent decision support system 400, rating an overall experienceusing the intelligent decision support system 400, and/orcollect/receive suggestions and/or comments. The displays 547 maycollect and/or store the feedback information in a database/memoryassociated with the intelligent decision support system 400 of FIG. 4and may use the feedback data to learn, train, and/or increase a degreeof accuracy for a prediction model.

Turning now to FIG. 5E, diagram 555 illustrates display 560 (e.g., auser interface/graphical user interface “GUI”) integrating a dialogsystem (e.g., chatbot) such as, for example, the dialog system 404 ofthe intelligent decision support system 400 of FIG. 4. In one aspect, byway of example only, the intelligent decision support service 402 (e.g.,dialog advisor) may be included in the display 560 and provide one ormore suggestions/recommendations. For example, thesuggestions/recommendations may include one or more factors that shouldbe included in a decision such as for example: compensationcompetitiveness, career potential and skills, additional informationfrom one or more associates (e.g., managers), and/or historical data(e.g., dates of previous promotions/compensation raises). Thus, thedialog system 404 of the intelligent decision support system 400illustrated in display 560 may include one or more employeequestions/queries and/or one or more compensation/salary programquestions/queries.

Turning now to FIG. 6 is a diagram 600 depicting operations of anintegrated dialog system complementing the intelligent decision supportsystem. In one aspect, many of the functional blocks previouslydescribed in FIGS. 1-5A-5E may be applied and used for executing one ormore operations and/or functionality described in FIG. 6. Repetitivedescription of like elements, components, modules, services,applications, and/or functions employed in other embodiments describedherein is omitted for sake of brevity.

The intelligent decision support system 400 using a dialog system 404 ofFIG. 4 may be a customized/personalized conversational advisor that mayaccess and retrieve personalized information 602 (e.g., employeeinformation), guidance data 604 (e.g., compensation/salary programguidance data), and issue a query using tool guidance 606.

The dialog system 404 may access the personalized information 602 thatmay include user information (e.g., employee information). The dialogsystem 404 may query a list of users/employees that may befetched/retrieved from a database. For example, the dialog system 404may issue a query for the personalized information 602 such as, forexample, “which employees did not get an increase in the past 2 years?”and “when was John Doe last promoted.” The dialog system 404 may issue aquery to the guidance data 604 such as, for example, “what factorsshould I consider for a salary increase?” or “how should I factor incheckpoint ratings.” The dialog system 404 may issue a query to the toolguidance 606 such as, for example, “what can I ask?” or “how does theintelligent decision support system 400 make suggestions?”

It should be noted that the present invention may use machine learningto combine different machine learning models to enhance the performanceof the individual models. In one embodiment, ensemble models may beused, which are a set of models whose individual predictions arecombined in a way that provides more accurate classification than theindividual models in the ensemble. For example, by using a ‘majorityvoting’ approach for combining the output of several constituent models,an ensemble model can ‘pick’ the more reliable models and ‘ignore’ theless reliable models for specific data points and thus increase theoverall accuracy.

Additionally, a decision tree may be a class discriminator thatrecursively partitions a training set until each partition consistsentirely or dominantly of records from the same class. The decision treemay have a root node, interior nodes, and multiple leaf nodes where eachleaf node is associated with the records belonging to a record class.Each non-leaf node of the tree contains a split point which is a test onone or more attributes to determine how the data records are partitionedat that node. Thus, as used herein, in the context of a selected domain(e.g., an HR domain), compensation decisions that are typically complexand are based on understanding different types of employee data (e.g.,skills, last increases, propensity to leave, performances, etc.), themachine learning model can assist a domain experts by capturingdifferent pieces of data and deriving an appropriate decision such as,for example, with respect to compensation where a compensation decisionis required based on data for each user/employee. A decision trees maybe employed to assist in a decision (e.g., high/medium/low-rateddecision) for each user/employee. An ensemble of multiple decision treesmay also be used to test the impact of different attributes/features.That is, a decision tree ensemble may be an ensemble classifier that mayinclude a collection of decision trees.

Turning now to FIG. 7A-7C, block diagrams 700, 710, and 720 depictoperations/steps for generating natural language reasons (“NLR”) forexplaining automatically generated machine learning recommendations froman ensemble of classifiers. In one aspect, many of the functional blockspreviously described in FIGS. 1-5A-5E may be applied and used forexecuting one or more operations and/or functionality described in FIGS.7A-7C. Repetitive description of like elements, components, modules,services, applications, and/or functions employed in other embodimentsdescribed herein is omitted for sake of brevity.

Starting in step 1 illustrated in FIG. 7A, a decision tree 702 may belinearized such as, for example, linearized, data tree 704, which may beperformed by the intelligent decision support system 400 of FIG. 1. Eachline in the linearized, data tree 704 may correspond to a single pathfrom a root leaf node of the decision tree.

In step 2 illustrated in FIG. 7B, the intelligent decision supportsystem 400 may map user data (e.g., employee data) with machine learning(“ML”) paths of the linearized, data tree 704. The intelligent decisionsupport system 400 may use as input an employee record 706 and adecision tree result. A record may be matched with the decision treepaths. The output 708 of the intelligent decision support system 400 maybe, for each user/employee, a set of machine learning rules that werefollowed in the path to reach a decision.

In step 3 illustrated in FIG. 7C, the output recommendations for each ofthe constituent decision trees of an ensemble for given data points(e.g., employee records) may be computed, as shown in block 712. In step4, one or more easily consumable natural language reasons (“NLRs”) 714corresponding to every unique machine learning rule in the pathsfollowed by any of the constituent decision trees of an ensemble toreach an output recommendation may be provided, as shown in block 714.Finally, the set of natural language reasons corresponding to the finalensemble recommendation for a data point is computed; in one instance,by selecting all the natural language reasons corresponding to the setof machine learning rules in the decision paths of all the decisiontrees that form the majority ‘vote’ in the ensemble of decision trees(e.g., majority voting with a tiebreaker rule may be used in thisinstance). The intelligent decision support system 400 may output, foreach user (e.g., employee), this computed set of natural languagereasons corresponding to the final ensemble recommendation, as shown inblock 716.

Thus, an intelligent decision support system may provide for generatinginterpretable natural language reasons from an ensemble of classifiers.When classifiers are decision trees, the present invention mayexplicitly detail a complete path of a decision tree taken by a machinelearning model to reach a decision. In domains where deeper and moreconsumable interpretability is desired (e.g., for HR decision making),the intelligent decision support system described herein may map ortranslate (e.g., using artificial intelligence and/or natural languageprocessing (“NLP”)) complete/entire machine learning path and thecorresponding decision into natural language reasons. The intelligentdecision support system may be used in applications such as, forexample, making a compensation decision where not only the decision butthe attribute values that contributed to the decision matter aredefined/translated.

Turning now to FIG. 8A, diagram 800 illustrates an intelligent decisionsupport system 800. As will be seen, many of the functional blocks mayalso be considered “modules” or “components” of functionality, in thesame descriptive sense as has been previously described in FIGS. 1-4.With the foregoing in mind, the module/component blocks of diagram 800may also be incorporated into various hardware and software componentsof a system for implementing an intelligent decision support system 810in accordance with the present invention. Many of the functional blocksof the intelligent decision support system 800 may execute as backgroundprocesses on various components, either in distributed computingcomponents, or on the user device, or elsewhere.

In one aspect, computer system/server 12, incorporating processing unit16 and memory 28 of FIG. 1 (not shown in FIG. 8 for illustrativeconvenience) may be used to perform various computational, dataprocessing and other functionality in accordance with various aspects ofthe present invention.

In one aspect, the intelligent decision support system 810 may include aconversation application programming interface (“API”) 802, a naturallanguage understanding (“NLU”) API 804, an orchestrator 806, a userinterface (“UP”) 808 and a database 812, each of which may be incommunication with each other. The intelligent decision support system810 may be in communication with user 820 (e.g., managers,administrators, supervisors, other employees, etc.).

In operation, the intelligent decision support system 810 may create acustomized/personalized conversational chatbots that can automaticallyanswer questions based on knowledge from unstructured informationsources and structured personal information, which may be stored indatabase 812. The knowledge/information in the structured informationsources may be customized/personal and dependent on the user such as,for example, user 820 asking the question after authentication. Theinformation being provided may be from an ontology/domain included inthe database 812 such as, for example, an HR domain and informationabout users/employees or guidance for managers. The personal informationmay be information about users/employees reporting to a user such as,for example, a user 820 that has accessed the intelligent decisionsupport system 810. The intelligent decision support system 810 mayrespond to one or more queries received from user 820.

More specifically, all queries may be sent to a natural languageunderstanding module (e.g., IBM® Watson® APIs 802), which can computethe underlying intent for each query and identify the entities andkeywords within each query and pass these on to the orchestrator 806.(IBM® and Watson® are trademarks of International Business MachinesCorporation). If the intent is personal information, the following maybe performed as shown in FIGS. 8A and 8B.

Turning now to FIG. 8B, an additional method 850 is illustrated forimplementing intelligent decision support in a computing environment(e.g., in a dialog system), in which various aspects of the illustratedembodiments may be implemented. The functionality 850 may be implementedas a method executed as instructions on a machine, where theinstructions are included on at least one computer readable medium or ona non-transitory machine-readable storage medium.

The functionality 850 may start in block 821 by receiving a user queryfrom a user and interpreting, providing, and/or analyzing the queryusing natural language understanding via a machine learning operation.In block 822, a type of query is determined such as, for example,determining if the query is for personal information or generalized(“generic”) information. If the type of query is for generalizedinformation, the method 850 may move to block 824 and retrieve one ormore answers to a query from a database (e.g., a question bank). Themethod 850 may then move to block 830 where the retrieved answers may beprovided back to the user (e.g., respond back to the user), as in block830.

Returning to block 822, the type of query is for personal information,the method 850 may move to block 826 and identify and/or resolved one ormore names. From block 826, the method 850 may move to block 828 andform a database query and retrieve results from the database, as inblock 828. The method 850 may, again, move to block 830.

Thus, in operations, operations of FIGS. 8A and 8B may be summarized asfollows. First, all the names in one or more queries may be identifiedbased on intents, the entities and keywords. For example, the identifiednames may be resolved against individuals for whom information is beingsought for and can be displayed (e.g., employees matching the name andreporting to the manager who has logged in and who is querying thechatbot). If multiple names match, a disambiguation strategy may beemployed to determine if the intent is for a list of users or for asingle user. If the query is for a single user and multiple names match,the chatbot may query back the user showing the set of matches andasking for a clarification of the exact individual. Once the individualor the list of individuals is determined, a database query (e.g., aStructured Query Language “SQL” query) may be formed to retrieve thedesired personal information about the specific individuals. Theinformation may be retrieved and shown to the user via the chatbot.

If the intent of the query is to obtain general information (e.g., anentity's program guidance, etc.), a response may be retrieved from aquestion bank and the UI may display the response back to user.

Turning now to FIG. 9, diagram 900 illustrates a display 910 (e.g., auser interface/graphical user interface “GUI”) providing recommendationsusing an intelligent decision support system using the intelligentdecision support systems 400 or 800 of FIGS. 4 and 8, which may use theintelligent decision support service 402 and the dialog system 404.

For example, the display 910 (e.g., the dialog system 404) may providethe ability to target a selected user (e.g., a targeted employee) formaking a decision such as, for example, determining a compensationadjustment using artificial intelligence (“AI”) basedadvice/suggestions. For example, one or more machine learningrecommendations may be provided for a compensation increaseprioritizations with supporting reasons/evidence via the display 910(e.g., customized/personalized chatbot/dialog system 404).

FIG. 10 is an additional block diagram depicting an intelligent decisionsupport system. As will be seen, many of the functional blocks may alsobe considered “modules” or “components” of functionality, in the samedescriptive sense as has been previously described in FIGS. 1-9. Withthe foregoing in mind, the module/component blocks of diagram 1000 mayalso be incorporated into various hardware and software components.

In operation, historical data 1006 (e.g., historical compensation datathat may include prior compensation increases history along with allavailable employee attributes at that time) may be collected. A separateannotated data set such as, for example the annotated data 1004 may becreated by having domain experts annotate data that could be syntheticdata or real data 1008 (e.g., pilot/real data) with fictional increasesmarked by one or more domain experts specifically for compensationincreases.

One or more different splits of historical data H may be derived. One ormore different splits of annotated data A may be derived. Data fields ofall the splits of historical data H and annotated data A may besynchronized. One or more common set of features may be derived based onuser attributes (e.g., employee attributes). It should be noted that thehistorical data and the annotated data (e.g., the historicalcompensations and collected annotated) may not be compatible. Thus, thehistorical data 1006 and the annotated data 1004 may be interpolated,and one or more different fields may be derived in the two datasets tohave comparable fields. For example, a last salary increase cycle in thehistorical data may be used to derive increment ratings to be predictedby a machine learning models and/or previous salary increases may beaggregated and organized in increments in last six months, last year,etc., to enable parallel timelines for different compensations. One ormore attributes may be derived using a combinations of fields.

One or more ensemble models 1016 (e.g., a majority ensemble model) formachine learning may be used using the splits of historical data H andannotated data A using models 1010, 1012, and 1014 (e.g., models 1,model 2, model 3). The individual models such as, for example, models1010, 1012, and 1014 provide complimentary insights. Thus, one or moresampling strategies on each datasets may be employed such as, forexample, a subset from historical data containing employees in annotatedand pilot sets, a subsample of historical data that has distributionsimilar to annotated data, combined subsamples of historical data withannotated data, and/or various sizes of samples. In one aspect, thepresent invention first constructs a set of weak models using abovementioned splits, and then combines them using one or more ensemblemethods, which leads to an aggregate strong model. For example, in oneinstantiation of invention, the present invention may use an ensemble ofthree different models and output is chosen as the majority among thethree classifiers.

In one aspect, one or more machine learning operations and/or decisiontrees may be used. A final classification 1018 (e.g., a final predictionon pilot/real data of salary decisions) may be generated using one ormore ensemble models 1016 (e.g., a majority ensemble model). To generatethe final classification, the classifications of the constituent modelsof the ensemble are combined using a combination technique (e.g., for agiven data point, picking/selecting the class with majority ‘votes’ froman individual models. Other combination techniques can also be used). Inone embodiment, a majority voting scheme may be used.

Thus, as described herein, the present invention may combine historicaldata/actions with new, domain expert annotation data to train one ormore machine learning models for an automated and intelligent decisionsupport system for helping with actions in a particular domain. In oneaspect, compatible semantics may be derived across historicalcompensations and current, data annotations and adjust for guidelineschanges. An array of prediction model may be generated using one or moredifferent splits of the data to compensate for much larger size ofhistorical data sets. An ensemble operation may be used combine theseprediction models to generate stronger and accurate decision models.

FIG. 11 is an additional block diagram depicting operations within anintelligent decision support system. As will be seen, many of thefunctional blocks may also be considered “modules” or “components” offunctionality, in the same descriptive sense as has been previouslydescribed in FIGS. 1-10. With the foregoing in mind, themodule/component blocks of diagram 1100 may also be incorporated intovarious hardware and software components.

That is, FIG. 11 illustrates 1100 operations for selecting samples oflabelled data (e.g., for a HR use case, employee data along withproposed salary increments) from data sets based on similarity withother data sets for building an intelligent decision support system suchas, for example, the intelligent decision support system 400 of FIG. 4.Repetitive description of like elements, components, modules, services,applications, and/or functions employed in other embodiments describedherein is omitted for sake of brevity.

In one aspect, data set A 1102 of labelled data (e.g., data labelledwith the decision being modelled such as, for example, employee dataalong with proposed salary increments for a HR use case) and data set B1104 of related/similar data (e.g., data annotated by domain experts togenerate examples of salary increase decisions for a HR use case) may beselected and/or used by a data selector 1106. For each data point B1from data set B 1104, the following operations may be performed. First,a closest/most similar match to any data point in data set A 1102 may beidentified using a distance measure comparing all attributes of datapoint (e.g., a Euclidean distance measure after converting allattributes to a numerical measure). To further illustrate, suppose theclosest match is to data point C (from data set A) and the correspondingdistance (between B1 and C) is D. Second, the corresponding distance Dmay be compared to an average distance between any two points in dataset A 1102 using a same distance measure.

Third, if the corresponding distance D is less than the averagedistance, the point B1 (from the data set B) may be added to the sampleset S 1108. Similarly, iterating through every data point in data set B,additional data points from the data set B may be added to the sampleset S 1108 if they are as similar to their respective closest match indata set A as the average similarity within data set A. Thus, a sampleset S, selected from data set B can be built that consists of all datapoints in B that are ‘similar’ to data points in data set A.

One or more machine learning models 1110 ML models may be built usingboth data set A and data set S to obtain increased/higher accuracymodels. In an additional aspect, other similarity measures such as, forexample, a Manhattan distance measure, may be used instead of Euclideandistance. Instead of comparing the distance D (e.g., for a data point B1from the data set B with the closest point C from the data set A) withthe average distance among data points in the data set A, in one otherembodiment, the comparison can be of D with the closest distance of anydata point in data set A with any other data point in data set A usingthe same distance measure. This will lead to a more conservativeselection of data points for the sample set S to consist of only thedata points in B that are really close to data points in A. In anotherembodiment, the comparison can be of D with the farthest distance of anydata point in data set A with any other data point in data set A usingthe same distance measure This will lead to a more liberal selection ofdata points for the sample set.

Thus, the present invention provides for selecting annotations (e.g.,labeled data) from one or more data sets based on a similarity with areference data set for use in a machine learning application tied to thereference data set. The selection of the annotations may be based onsimilarity measures between data points in the reference data set andexamples of other data sets. Additionally, the selection of theannotations may be based on a comparison of the closeness/similarity ofa data point in other data sets with any point in the reference data setand a comparison of this with the closest or average or farthestdistance between any two points in the reference data set (differentembodiments). The similarity/distance measure may be a Euclideandistance or Manhattan distance. The present invention provides anapplication for decision support systems where there is a data set ofhistorical decisions and a data set of newly annotated data (referencedata set) for specific decisions.

Turning now to FIG. 12, an additional method 1200 is illustrated forimplementing intelligent decision support in a computing environment(e.g., in a dialog system), in which various aspects of the illustratedembodiments may be implemented. The functionality 1200 may beimplemented as a method executed as instructions on a machine, where theinstructions are included on at least one computer readable medium or ona non-transitory machine-readable storage medium.

The functionality 1200 may start in block 1202. Similar data may becombined/collected according to historical data and annotated data, asin block 1204. One or more prediction models may be generated usingdifferent splits of the similar data, as in block 1206. The one or moreprediction models may be combined to generate a decision model using oneor more machine learning operations, as in block 1208. The functionality1200 may end in block 1210.

In one aspect, in conjunction with and/or as part of at least one blockof FIG. 12, the operations of 1200 may include each of the following.The operations of 1200 may combine structured and unstructured data fromone or more data sources and customize communications in a dialog systemaccording to the structured and unstructured data. The historical dataand the annotated data may be interpolated using an ensemble ofclassifiers.

The operations of 1200 may select the annotated data from a data setbased on a degree of similarity with one or more alternative data sets.The operations of 1200 may explain one or more recommendations from theone or more prediction models according to a natural language operation.

In one aspect, the operations of 1200 may collect the historical data,collect the annotated data from one or more domain experts, derive thedifferent splits of both the historical data and the annotated data,and/or determine one or more features from one or more attributes of aselected entity from similar data fields of the different splits.

The operations of 1200 may initiate a machine learning to perform one ormore machine learning operations to perform a semantic analysis, train aclassifier, learn one or more machine learning rules, learn contextualdata associated with the dialog system, learn and train the one or moreprediction models using the historical data and the annotated data,generate one or more recommendations or predictions from the one or moreprediction models, and assist with engaging in communication using thedialog system, or perform a combination thereof.

Turning now to FIG. 13, an additional method 1300 is illustrated forimplementing intelligent decision support in a computing environment(e.g., in a dialog system), in which various aspects of the illustratedembodiments may be implemented. The functionality 1300 may beimplemented as a method executed as instructions on a machine, where theinstructions are included on at least one computer readable medium or ona non-transitory machine-readable storage medium.

The functionality 1300 may start in block 1302 by collecting data aboutcollect data about historical data (e.g., historical decisions), as inblock 1304. Annotated data (e.g., examples of decisions) may becollected/generated from one or more domain experts, as in block 1306.One or more machine learning models may be generated/used usingdifferent splits of the historical data and the annotated data togenerate recommendations for the decisions, as in block 1308. The one ormore machine learning models may be used to generate finalrecommendations for a decision using one or more ensemble machinelearning models, as in block 1310. A dialog system may be activated forinteracting with one or more users for providing data for decisionmaking, the final recommendations, reasons for recommendations and forengaging in a dialog with the one or more users relating to the data,the decision making, the final recommendations, reasons forrecommendations, as in block 1312. The functionality 1300 may end inblock 1314.

In one aspect, in conjunction with and/or as part of at least one blockof FIG. 13, the operations of 1300 may include each of the following.The operations of 1300 may, using the dialog system, display structuredand unstructured data of one or more entities (e.g., users, business,persons, organizations, etc.) and information about decision making fromone or more data sources. The operations of 1300 may provide analyticsdynamically analyzing domain expert decisions, analytics suggestingevidence and reasons for the recommendations. The operations of 1300may, using the dialog system, provide a natural language conversationaldialog system for answering user queries about the data aboutindividuals, information about decision making, recommendations, andreasons behind recommendations, an ability for users to enter orvalidate data about individuals shown to them; and grouping of allemployee attributes to reflect business priorities for the correspondingdecisions.

The operations of 1300 may automatically determine the intendedindividual or group of individuals about whom information is requestedby users interacting with the conversational dialog system using naturallanguage processing, automatically retrieve the information requested byusers about the individual or group of individuals, and/or display/showthe specific data about specific individual or individuals to usersinteracting with the conversational dialog system after authenticationdepending on the information sharing policies in effect and the specificqueries by the users; and additionally, showing generic data applicableto all users to users interacting with the conversational dialog systemdepending on the information sharing policies in effect and the specificqueries by the users.

The operations of 1300 may select samples of data sets to be used bymachine learning models based on a degree of similarity with a referencedata set and using the sampled data set as additional data for trainingthe machine learning models.

The operations of 1300 may determine the similarity (e.g., degree ofsimilarity) between data points in the reference data set and datapoints in the data sets being sampled, determine the similarity betweenall possible pairs of data points in the reference data set; determinethe smallest, largest and average values of the similarity between allpossible pairs of data points in the reference data set, and/or selectdata points from the data sets being sampled for inclusion in the sampledata set if the computed degree of similarity measures for these datapoints is lower than the smallest or largest, or average values of thedegree of similarity between any two data points in the reference dataset. The degree of similarity measure used for measuring the similaritybetween two data points is one or more of the following measures: 1) thesum of the number of attributes that match exactly between two datapoints, 2) the cosine distance between the numerical representations ofthe two data points, 3) the Euclidean distance between the numericalrepresentations of the two data points, and/or 4) the Manhattan distancebetween the numerical representations of the two data points.

The operations of 1300 may generate interpretable natural languagereasons for the recommendations generated by one or more machinelearning models and displaying the reasons along with therecommendations in a user interface (e.g., using a dialog system) withusers of the decision support system.

The operations of 1300 may perform one or more of the following when themachine learning models are decision trees by: 1) computing the completeset of machine learning rules that were followed in the path by thedecision tree to arrive at an output recommendation; 2) mapping everyunique machine learning rule in the path to an easily consumable naturallanguage reason; and/or 3) displaying/showing the set of naturallanguage reasons as the reasons for a recommendation by the machinelearning model to the user via the user interface.

The operations of 1300 may perform one or more of the following themachine learning models are ensembles of decision trees by: 1) for everyconstituent decision tree model in the ensemble, computing the completeset of machine learning rules that were followed in the path by thatdecision tree to arrive at an output recommendation; 2) mapping everyunique machine learning rule in the paths by all of the constituentdecision tree models to an easily consumable natural language reason; 3)for each data point for which recommendations are being generated,creating a set of reasons by computing a union of the set of naturallanguage reasons mapped from the paths followed by the subset ofconstituent decision tree models that participated in the finalrecommendation by the ensemble of decision trees for that data point,and/or 4) for each data point for which recommendations are beinggenerated, showing this set of natural language reasons as the reasonsfor the recommendation by the ensemble machine learning model to theuser via the user interface.

The operations of 1300 may perform collect the historical data; generatethe annotated data using one or more human domain experts; derivecompatible semantics across historical data and annotated data adjustingfor differences; derive different splits of the historical data and theannotated data; train different machine learning models for generatingrecommendations using the different splits; and/or create one or moreensemble machine learning models combining the recommendations from eachof the different machine learning models to generate the finalrecommendations to be shown to users via the user interface.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowcharts and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowcharts and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowcharts and/or block diagram block orblocks.

The flowcharts and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowcharts or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustrations, and combinations ofblocks in the block diagrams and/or flowchart illustrations, can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts or carry out combinations of special purposehardware and computer instructions.

What is claimed is:
 1. A method for implementing intelligent decisionsupport for decision making in a computing environment by a processor,comprising: collecting data about a plurality of historical data;generating one or more samples of annotated data by one or more domainexperts; generating one or more machine learning models using differentsplits of the data and the annotated data to generate recommendationsfor one or more decisions; combining the one or more machine learningmodels to generate a recommendation for the one or more decisions usingone or more ensemble machine learning models; generating one or morenatural language reasons for the recommendations by one or more machinelearning models and providing one or more reasons for the recommendationusing a dialog system and performing each of the following upon the oneor more machine learning models being an ensemble of decision trees: foreach constituent decision tree model in the ensemble, determining a setof machine learning rules that were used in a path by a decision tree toarrive at an output recommendation; mapping each machine learning rulein the path by each of the constituent decision tree models to a naturallanguage reason; for each data point for which the recommendations arebeing generated, creating a set of reasons by computing a union of a setof natural language reasons mapped from paths followed by a subset ofthe constituent decision tree models that participated in a finalrecommendation by the ensemble of decision trees for the data point; andfor each of the data points for which recommendations are beinggenerated, displaying the set of natural language reasons as the reasonsfor the recommendation by the ensemble machine learning model; andinteracting with a user via the dialog system and engaging in a dialogwith the user in relation to the questions about the data and therecommendation.
 2. The method of claim 1, further including: displayingstructured and unstructured data via the dialog system pertaining to oneor more entities and information from one or more data sources;displaying an analysis of decisions up to a current time period;providing evidence for the one or more reasons for the recommendations;engaging in a series of queries with the user in relation to the data,the recommendation, or a combination thereof; enabling the user to enteror validate data; and grouping one or more attributes of one of aplurality of entities to reflect one or more priorities of an entityrelating to the data, the recommendation, or a combination thereof. 3.The method of claim 1, further including: automatically determining anintended entity or group of entities pertaining to requested informationby the user via the dialog system; automatically retrieving therequested information about the intended entity or the group ofentities; identifying specific data about the intended entity or groupof entities according to one or more policies in effect and specificqueries by the users; and providing generic data applicable to aplurality of users interacting with the dialog system according to theone or more policies in effect and specific queries by the user.
 4. Themethod of claim 1, further including: selecting sampled data sets to beused by machine learning models based on a degree of similarity with areference data set and using the sampled data set as additional data fortraining the machine learning models; determining the degree ofsimilarity between data points in the reference data set and the datapoints in the sampled data sets; determining the degree of similaritybetween all pairs of data points in the reference data set; determininga smallest value, a largest value, and an average values of the degreeof similarity between the pairs of data points in the reference dataset; and selecting the data points from the sampled data sets if thedegree of similarity is lower than the smallest value, the largestvalue, or the average values of the degree of similarity between any twodata points in the reference data set.
 5. The method of claim 1, furtherincluding: collecting the historical data; generating the annotated datausing one or more domain experts; deriving compatible semantics acrosshistorical data and the annotated data adjusting for differences;deriving different splits of the historical data and the annotated data;training the machine learning models for generating one or morerecommendations using the different splits; and creating ensemblemachine learning models by combining the one or more recommendationsfrom each of the machine learning models to generate the recommendation.6. The method of claim 1, further including initiating a machinelearning to perform one or more machine learning operations to perform asemantic analysis, train a classifier, learn one or more machinelearning rules, learn contextual data associated with the dialog system,learn and train the one or more prediction models using the historicaldata and the annotated data, generate one or more recommendations orpredictions from the one or more prediction models, and assist withengaging in communication using the dialog system, or perform acombination thereof.
 7. A system for implementing intelligent decisionsupport for decision making in a computing system, comprising: one ormore computers with executable instructions that when executed cause thesystem to: collect data about a plurality of historical data; generateone or more samples of annotated data by one or more domain experts;generate one or more machine learning models using different splits ofthe data and the annotated data to generate recommendations for one ormore decisions; combine the one or more machine learning models togenerate a recommendation for the one or more decisions using one ormore ensemble machine learning models; generate one or more naturallanguage reasons for the recommendations by one or more machine learningmodels and providing one or more reasons for the recommendation using adialog system and perform each of the following upon the one or moremachine learning models being an ensemble of decision trees: for eachconstituent decision tree model in the ensemble, determining a set ofmachine learning rules that were used in a path by a decision tree toarrive at an output recommendation; mapping each machine learning rulein the path by each of the constituent decision tree models to a naturallanguage reason; for each data point for which the recommendations arebeing generated, creating a set of reasons by computing a union of a setof natural language reasons mapped from paths followed by a subset ofthe constituent decision tree models that participated in a finalrecommendation by the ensemble of decision trees for the data point; andfor each of the data points for which recommendations are beinggenerated, displaying the set of natural language reasons as the reasonsfor the recommendation by the ensemble machine learning model; andinteract with a user via the dialog system and engaging in a dialog withthe user in relation to the questions about the data and therecommendation.
 8. The system of claim 7, wherein the executableinstructions: display displaying structured and unstructured data viathe dialog system pertaining to one or more entities and informationfrom one or more data sources; display an analysis of decisions up to acurrent time period; provide evidence for the one or more reasons forthe recommendations; engage in a series of queries with the user inrelation to the data, the recommendation, or a combination thereof;enable the user to enter or validate data; and group one or moreattributes of one of a plurality of entities to reflect one or morepriorities of an entity relating to the data, the recommendation, or acombination thereof.
 9. The system of claim 8, wherein the executableinstructions: automatically determine an intended entity or group ofentities pertaining to requested information by the user via the dialogsystem; automatically retrieve the requested information about theintended entity or the group of entities; identify specific data aboutthe intended entity or group of entities according to one or morepolicies in effect and specific queries by the users; and providegeneric data applicable to a plurality of users interacting with thedialog system according to the one or more policies in effect andspecific queries by the user.
 10. The system of claim 8, wherein theexecutable instructions: select sampled data sets to be used by machinelearning models based on a degree of similarity with a reference dataset and using the sampled data set as additional data for training themachine learning models; determine the degree of similarity between datapoints in the reference data set and the data points in the sampled datasets; determine the degree of similarity between all pairs of datapoints in the reference data set; determine a smallest value, a largestvalue, and an average values of the degree of similarity between thepairs of data points in the reference data set; and select the datapoints from the sampled data sets if the degree of similarity is lowerthan the smallest value, the largest value, or the average values of thedegree of similarity between any two data points in the reference dataset.
 11. The system of claim 8, wherein the executable instructions:collect the historical data; generate the annotated data using one ormore domain experts; derive compatible semantics across historical dataand the annotated data adjusting for differences; derive differentsplits of the historical data and the annotated data; train the machinelearning models for generating one or more recommendations using thedifferent splits; and create ensemble machine learning models bycombining the one or more recommendations from each of the machinelearning models to generate the recommendation.
 12. The system of claim8, wherein the executable instructions initiate a machine learning toperform one or more machine learning operations to perform a semanticanalysis, train a classifier, learn one or more machine learning rules,learn contextual data associated with the dialog system, learn and trainthe one or more prediction models using the historical data and theannotated data, generate one or more recommendations or predictions fromthe one or more prediction models, and assist with engaging incommunication using the dialog system, or perform a combination thereof.13. A computer program product for, by a processor, implementingintelligent decision support for decision making in a computing system,the computer program product comprising a non-transitorycomputer-readable storage medium having computer-readable program codeportions stored therein, the computer-readable program code portionscomprising: an executable portion that collects data about a pluralityof historical data; an executable portion that generates one or moresamples of annotated data by one or more domain experts; an executableportion that generates one or more machine learning models usingdifferent splits of the data and the annotated data to generaterecommendations for one or more decisions; an executable portion thatcombines the one or more machine learning models to generate arecommendation for the one or more decisions using one or more ensemblemachine learning models; an executable portion that generates one ormore natural language reasons for the recommendations by one or moremachine learning models and providing one or more reasons for therecommendation using a dialog system and an executable portion thatperforms each of the following upon the one or more machine learningmodels being an ensemble of decision trees: for each constituentdecision tree model in the ensemble, determining a set of machinelearning rules that were used in a path by a decision tree to arrive atan output recommendation; mapping each machine learning rule in the pathby each of the constituent decision tree models to a natural languagereason; for each data point for which the recommendations are beinggenerated, creating a set of reasons by computing a union of a set ofnatural language reasons mapped from paths followed by a subset of theconstituent decision tree models that participated in a finalrecommendation by the ensemble of decision trees for the data point; andfor each of the data points for which recommendations are beinggenerated, displaying the set of natural language reasons as the reasonsfor the recommendation by the ensemble machine learning model; and anexecutable portion that interacts with a user via the dialog system andengaging in a dialog with the user in relation to the questions aboutthe data and the recommendation.
 14. The computer program product ofclaim 13, further including an executable portion that: displaysstructured and unstructured data via the dialog system pertaining to oneor more entities and information from one or more data sources; displaysan analysis of decisions up to a current time period; provides evidencefor the one or more reasons for the recommendations; engages in a seriesof queries with the user in relation to the data, the recommendation, ora combination thereof; enables the user to enter or validate data; andgroups one or more attributes of one of a plurality of entities toreflect one or more priorities of an entity relating to the data, therecommendation, or a combination thereof.
 15. The computer programproduct of claim 13, further including an executable portion that:automatically determines an intended entity or group of entitiespertaining to requested information by the user via the dialog system;automatically retrieves the requested information about the intendedentity or the group of entities; identifies specific data about theintended entity or group of entities according to one or more policiesin effect and specific queries by the users; and provides generic dataapplicable to a plurality of users interacting with the dialog systemaccording to the one or more policies in effect and specific queries bythe user.
 16. The computer program product of claim 15, furtherincluding an executable portion that: selects sampled data sets to beused by machine learning models based on a degree of similarity with areference data set and using the sampled data set as additional data fortraining the machine learning models; determines the degree ofsimilarity between data points in the reference data set and the datapoints in the sampled data sets; determines the degree of similaritybetween all pairs of data points in the reference data set; determines asmallest value, a largest value, and an average values of the degree ofsimilarity between the pairs of data points in the reference data set;and selects the data points from the sampled data sets if the degree ofsimilarity is lower than the smallest value, the largest value, or theaverage values of the degree of similarity between any two data pointsin the reference data set.
 17. The computer program product of claim 13,further including an executable portion that: collects the historicaldata; generate the annotated data using one or more domain experts;derive compatible semantics across historical data and the annotateddata adjusting for differences; derive different splits of thehistorical data and the annotated data; train the machine learningmodels for generating one or more recommendations using the differentsplits; create ensemble machine learning models by combining the one ormore recommendations from each of the machine learning models togenerate the recommendation.
 18. The computer program product of claim15, further including an executable portion that initiates a machinelearning to perform one or more machine learning operations to perform asemantic analysis, train a classifier, learn one or more machinelearning rules, learn contextual data associated with the dialog system,learn and train the one or more prediction models using the historicaldata and the annotated data, generate one or more recommendations orpredictions from the one or more prediction models, and assist withengaging in communication using the dialog system, or perform acombination thereof.